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Abstract 


Electronic publication (EP) is a complex product, which integrates the achievements of modern 
technology, content of the subject area and teaching methods, as well as design and artistic 
qualities. The transition from book reviewing to electronic publication reviewing is similar to the 
exit from the plane into multidimensional space. It’s only natural that pedagogical, 
methodological and creative possibilities are much higher there, but it makes determining the 
technology and criteria for quality evaluation more difficult. 
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1. Introduction 


In modern society education’s informatization covers all areas of educational activity, 
therefore the electronic educational materials’ creation and use is an integral part of the 
educational process. Their use creates a comfortable environment for learning in a youth-friendly 
computer environment. Evaluation system of electronic publications is based on requirements for 
them as well as review of functional properties compared to software capabilities and technical 
properties of the publication. 


2. Analysis of recent studies and publications 


The large-scale work on the creation of electronic educational resources which is being 
carried out at universities today requires not only forming the teacher’s corresponding ICT 
competences, but also identifying significant criteria and creating the expert evaluation 
methodology of the developed educational electronic publications’ quality using these criteria. 


Most of Ukrainian and Western researchers’ works are only devoted to electronic 
publications’ classification study, which does not guarantee the creation of a high-quality 
educational resource. 


In this regard research works of D. Fedasyuk, N. Hotz, N. Mikiychuk, V. Kuts and G. 
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Kravtsov are relevant. However, there are no studies to date where quality evaluation of electronic 
educational publications in the ICT area is proposed. 


It is important that developing international standards for information and 
communication of education lies in cooperation between the International Organization for 
Standardization and other international organizations and committees, including the 
International Electrotechnical Commission. International standard ISO / IEC 19796: 2005 
Information technology. Learning, education and training. Quality management, assurance 
and metrics is the basis for description, comparison, approaches to analysis, quality management 
and quality assurance in this area. It serves as a tool for comparing existing approaches and their 
coordination using common quality model as a foundation. A key element of the standard is the 
Reference Framework for the description of qualitative approaches. 


The ISO / IEC 19796: 2005 standard consists of three parts: 


1) ISO / IEC 19796-1: 2005 — Part 1. The general approach is the first step 
towards the creation of a harmonized education model based on information 
technology; 


2) ISO / IEC 19796-2 — Part 2: Harmonized Quality Model identifies tools 
and indicators to implement a common approach to quality; 


3) ISO / IEC 19796-3 2009 — Part 3. Reference methods and metrics (which 
was mentioned in the previous part ISO / IEC 19796-1: 2005) extends the 
boundaries of standard structure for describing the approaches to quality. 
This is done by introducing a coherent description of recommended methods 
and metrics to implement the quality assurance and quality management 
system. Stakeholders involved in design and development use IT systems for 
education and learning. 


But, unfortunately, most of ISO / IEC 19796: 2005 requirements only apply to 
standards of design quality as well as development quality and use of IT systems for education and 
learning quality. Therefore we can say that international standards also do not disclose the concept 
of the electronic educational resources quality as well as they don’t include requirements to ensure 
it. As mentioned above, implementation of electronic educational resources into educational 
processes is a global trend today. One can find various methods for their quality control in the 
literature. 


In general, their disadvantages are: 


- Forming an excessive number of requirements which complicates 
evaluating and determining how satisfactorily assessment; 


- Duplicating content requirements; 


- The presented methods of monitoring and testing do not systematize 
requirements; 


- Structural expertise methods allow evaluating the structure only, as it 
doesn’t allow evaluating objective, methodological and scientific content of 
electronic educational resources; 


- Complexity of estimation algorithms. 


Yet, all of these methods are aimed at one goal - control of electronic educational 
resources conformity to specific requirements. This process is called certification in the technical 
regulations. This is a third-party action that proves that a properly identified product, process or 
service is in accordance with a specific standard or other regulatory document. 
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textbooks. 


Therefore the following is desirable: 


i For ensuring and quality control it is necessary to formulate a list of 
requirements for electronic educational resources and develop a 
methodology for internal certification of electronic educational resources at 
the university in accordance with existing certification rules. 


2 To implement the comparability of various components of electronic 
educational resources, a methodology for assessing their quality level based 
on the theory of qualimetry as an assessment of the conformity degree of the 
electronic educational resource with the established requirements is 
necessary. 


The purpose of the work is developing criteria for evaluating the quality of the 
educational electronic publication and determining the learning and didactic advantages as well 
as their implementation in the educational process in comparison with the classical school 


Tasks that need to be solved in order to achieve the goal: 


- Identifying the stages of designing an electronic educational publication 
and requirements of electronic publication quality; 


- Developing a system of indicators of an electronic educational publication 
quality, which integrates didactic, technical, design-ergonomic and 
technological characteristics; 


- Evaluating electronic publications with the proposed criteria and 
analyzing the results. 


3, Presentation of main material E-publication development 


In structure of electronic textbook it is possible to highlight at least three components: 
theoretical blocks, model blocks and knowledge control blocks. 


1 Theoretical blocks bear the main information load. It is better to avoid common 
situations when information blocks are formed using mere scan of book editions. In 
most of such cases not only the electronic version isn’t better, but it also significantly 
loses to the paper version due to combination of printed text perception 
disadvantages and lacks of perception of static information from the monitor screen. 
A team of software developers together with methodologists should choose the most 
effective ways of presenting the learning material depending on its content. 
According to the authors, successful variant of the theoretical block includes voice 
presentation of the material accompanied by dynamic illustrations on the computer 
display. Ensuring internal communication in the presentation can be implemented 
more effectively using hypertext — the Web-based technology. 


2 Model blocks — models of processes or phenomena. Models usage in the 
educational process is especially relevant, as we can’t carry out a more or less 
significant real experiment. An idealized model of a real process or phenomenon is 
created as a program piece that makes it possible to change the parameters of this 
process. 


Subprogram models can be created by manufacturer (internal models are embedded 
in the textbook while external are connected) as well as by the user of software 
system, e.g. by a teacher. A third option is possible — connecting models which were 
created by third-party manufacturers. The most widespread option to date are 
internal models which greatly simplifies the internal architecture of such textbooks. 
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However, the negative consequences of this approach lies in the impossibility of 
textbook’s flexible adaptation to real conditions of the educational process, which is 
an obvious disadvantage. 


3. Knowledge control blocks include list of questions and modules for analyzing 
answers (rating), selecting the appropriate option for further actions (depending on 
the evaluation, repeating the previous topic or transiting to the next one can be 
chosen). The questionnaire is implemented as an opened, closed or combined test. 
It is obvious that all of the above related to dynamic configuration is fully applicable 
to the knowledge control blocks. 


Summarizing the above, let's bring in the factors’ diagram that helps establishing the 
classification of electronic publications in terms of content, technological base, complexity and 
other aspects. 


By content 


By comple- " = 

teness of By purpose 
| using forms of and target 
| 


semantical audience 
information 


Electronic 


publications 


By struc- \, 
ture of 
user- 
system 
interface 





By software 
usage 


By techno- 
logical base 
(hardware and 
output) 


Figure 1 — Classification principles 


4. Criteria for quality evaluation of electronic publication 


When developing quality indicators system for online EEP, it should be taken into 
account that such resources do not have a “rigid” final version of structure and content and they 
are constantly changing throughout their life cycle while adapting to changeable requirements of 
the educational process and learning outcomes. Therefore, in addition to the set of indicators that 
are commonly used in evaluation of EEP (or electronic educational publications), indicators which 
allow evaluating effectiveness of the online EEP and its relevance (updating the database of 
reference materials, learning and methodical support, etc.) should be examined. In order to 
evaluate the effectiveness of an online EEP, indicators that are commonly used when evaluating 
web sites can be used: usability, attendance, time spent on a site, popularity / lack of demand of 
individual pages of a site, etc. 


The system of quality evaluation indicators for online EEP by stages of the life cycle is 
presented in Table 1. Indicators are measured on a four-point scale, where “3” is a better 
representation of the current situation, according to the expert, and “o” is the absence of the 
indicator. 


For accounting the significance of individual indicators, a ranking procedure based on 
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one of expert methods (e.g., hierarchy analysis method, paired comparisons method etc.) can be 
used in the final integrated evaluation of indicators. The result of ranking is a vector of weight 
coefficients of indicators and groups of indicators, which will increase the accuracy of evaluation. 


Thus, the resulting quality evaluation will be defined as a convolution of indicators. 


Table 1 — Description of EEP quality indicators 


Criteria description by points 
1 STAGE OF EEP DESIGNING 
1.1 Structural-Target Indicators Group 


3 - Presence of a course’s work program, defined requirements for the 

pass level of competency and results of learning, expressed in the 

form of formed competencies as a separate element of the EEP. 

1.1.1 Presence and Furthermore, requirements are detailed in the methodology for 

reasonableness of the certain types of training work and independent work. 
purpose 2 - Presence of a course’s work program, requirements to the pass 
level of competency and results of learning are partly determined. 
1 - Presence of a work program. 
o - There are no materials for this component. 


3 - EEP has a clear structure corresponding to the logic of studying 
the material as well as corresponding to the course work program. 
The structure displays all kinds of topics and educational work to be 
performed. 
1.1.2 The modular 2 - EEP has a modular structure, but it does not fully display the logic 
structure of the EEP of studying the material and the composition of the work to be 
performed. 
1 - EEP is structured, but not by the modular principle of material 
studying. 
o - EEP is not structured. 


3 - The high quality of the presented educational materials, EEP is 
developed on the basis of its own scientific, educational, 
1.2.1 Scientific and methodological developments of the author. References to these 
newness of the presented developments are presented in EEP. 
educational-methodical | 2 - The presented material is relevant and it matches scientific and 
material technical level in the analyzed subject area. 
1 - Materials presented in EEP are references to external works. 0 - 
No material available. 


Criteria description by points 
1.2 The group of indicators that describe the content quality 
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3 - EEP materials are set out with language the target audience 
understands, contain enough illustrative material (tables, figures 
etc.), are structured good enough and they highlight the main ideas 
1.2.2 Clearness and : ang os ee . 
understandability of 2- EEP materials are presented in a language accessible to the target 
: audience, but not illustrated sufficiently, main moments are not 
material eagiae 
highlighted. 
1 - Materials are presented in a manner of speech that does not match 
he target audience and are not illustrated and structured sufficiently. 
o - No material available. 


3 - Presentation of EEP materials includes practical examples, 
explanation of complex moments, highlighting the most difficult-to- 
understand questions, additional explanations and interdisciplinary 

connections. 
1.2.3 Problematique of | 2 - The presented material contains practical examples, but it does not 
material presentation display interdisciplinary connections with other fields of study. 
1 - Insufficient number of examples in the educational materials of 
EEP, interdisciplinary connections with other fields of study are not 
displayed. 
o - No material available. 


3 - EEP contains educational and methodical materials for all types o 
work and for all modules. 
1.2.4 Sufficiency of 2-EEP doesn’t contain educational and methodological materials for 
Saaeabonal pee all the main types of work that are set out by work program of the 
méthodslogical materials course (excluding independent work). 
1- EEP doesn’t contain educational and methodological materials for 
all types of work that are set out by work program of the course. 
o - No material available. 


1.3 Group of EEP’s Informational, Methodological and Organizational Support Indicators 


3 - EEP presents different forms and variants of interim and final 
tasks depending on their complexity level, and there is an 
explanation to evaluation of the final results. 
1.3.1 Taking students’ 2 - EEP presents different variants of interim and final tasks 
individual characteristics | depending on their complexity level, and there is an explanation to 
into account evaluation of the final results. 
1- EEP presents different variants of interim and final task which are 
not differentiated by complexity levels. 
o - EEP presents one education trajectory only. 
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3 - EEP activates interest in obtaining new knowledge, necessitates 
work with different types and forms of educational material, is 
interactive and multimedia, contains interesting links and materials 
related to the subject area and provides a system for encouraging 
students to study successfully. 

2 - EEP presents interesting links and materials related to the subject 
area. It also provides a system for encouraging students to study 
successfully. 

1 - There is a system for encouraging students to study successfully. 

o — EEP doesn’t encourage students to study. 


Criteria description by points 


3 - EEP provides informational and methodological instructions to all of 
EEP modules with explanation of right order of studying the material as 
1.3.3 Availability and well as explanations of tasks’ arrangement, of work with literature etc. 
sufficiency of Materials are presented as separate elements of the EEP. 
informational and 2 - EEP provides informational and methodological instructions to 
methodological materials some of the modules and tasks of EEP, but they are marked as 
as well as explanations for separate elements of EEP. 
different types of 1 - EEP provides informational and methodological instructions for 
educational activities | EEP’s individual elements studying that are not marked as separate 
elements of EEP. 
o - Informational and methodological materials are not presented. 


1.3.2 Increasing 
motivation to study 


3 - The list of literature, a glossary and links to Internet sources are 
1.3.4 Sufficiency of ‘ , presented in the EEP. ‘ 
supporting material 2 - The list of literature and a glossary are presented in EEP. 
1 - The list of literature is presented in EEP. 
o - Additional materials are not presented. 


1.4. Group of indicators that characterize EEP control functions 


3 - Different types and forms of intermediate and final control, 
including interactive forms of control, are presented in the 
corresponding LMS system (essays, tests, seminars) which is used in 
EEP. Control forms display the specifics of tasks. 

2 - Different types and forms of interim and final control are used in 
EEP, but EEP still lacks interactive forms. 

1- The forms of control used in EEP are of the same type, they do not 
display the specifics of tasks. Interactive forms of control are not 
used. 

o - Such component is not presented. 


1.4.1 Variety of control 
and self-control forms 


3 - Diagnosis of knowledge level at the beginning, various types of 
tasks’ completion results and the final control’s results is carried out 
1.4.2 Diagnosis of : . F Tne oS : ; 
iearnine outcomes 2 - Diagnosis of certain types of tasks completion results, as well as 
the final control’s results is carried out in EEP. 
1 - Diagnosis of the final control’s results is carried out in EEP. 
o - Such component is not presented. 
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3 - Sufficient supervisory tasks for understanding and mastering of 
Peer educational material, the number of test tasks exceeds 200 units. 
1.4.3 Availability of ae : . ; 
2 - Sufficient supervisory tasks for understanding and mastering of 
control and measurement ral sal th bevot k d : 
Racaale educational material, the number of test tasks exceeds 100 units. 
1 - EEP only has test tasks and their number exceeds 100 units. 
o - Control and measurement materials are not sufficient. 


2 STAGE OF EEP REALIZATION 
2.1 Group of design-ergonomic indicators 


3 - High quality of text material. Colors, fonts and similar methods of 
text highlighting in accordance with the logic of presentation. Text 
design is coherent, the text is readable, and possibility of 
2.1.1 Quality of text personalization is provided. 
material 2 - Good quality design of text material, text design is coherent and 
text is readable. 
1 - Low quality design of text material, text is not structured. 
o - Text is unreadable. 


Criteria description by points 


3 - High quality of graphic, audio and video content design. EEP has 


2.1.2 Quality and sufficient quantity of such content for students to explore. 


sufficiency of graphic |2- Good quality of graphic, audio and video content design. EEP has 


images, audio and video insufficient or excessive quantity of such content. 
material 1 - Low quality of graphic, audio and video content design. 
Oo - Materials are presented. 


2.2 Group of technical indicators 


3 - All of EEP links are functioning. 
2.2.1 Links workability 2 - EEP contains insignificant number of invalid links. 
a 1 - EEP contains significant number of invalid links. 
0 - Most of EEP links are invalid. 


Criteria description by points 


3 - Different formats are available for viewing educational materials. 
EEP materials are presented using modern free circulating formats, 
their viewing does not require additional software (or its installation 
2.2.2 Convenience of is not hard and a corresponding manual is provided). 
materials presentation 2 - EEP materials are presented using modern free circulating 
based on using different | formats, their viewing does not require additional software (or its 
electronic formats installation is not hard and a corresponding manual is provided). 
1 - While viewing some of EEP materials problems related to selected 
format of presentation may arise. 
0 - Materials are visible. 


3 STAGE OF EEP EXPLOITATION 
3.1 Indicators of EEP supporting 
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3 - All of students’ works (including tests and other types of control) 
are checked in time. Schedule of checkpoints is presented. Records 
3.1.1 Journaling of are kept in a grade book and students have access to it. 
students’ academic 2 - Tests are the only checkpoints. Records are kept in a grade book. 
achievements Schedule of checkpoints is presented. 
1 - Tests are the only checkpoints. Records are kept in a grade book. 
o - Checkpoints are not defined. 


Criteria description by points 


3.1.2 Support of feedback} 3 - Students are informed about upcoming events in time (via the 
with students (sending forum or sending personal messages). Teacher answers students’ 
messages and answering questions and moderates the forum. 
messages as well as 2 - Teacher answers students’ questions and informs them. 
informing about 1 - Teacher answers students’ questions. 
upcoming events) o - Teacher does not support feedback. 





5. Results of experimental work 


The considered indicators of EEP quality evaluation were tested in the educational 
process of Kharkiv comprehensive school N° 131 for evaluating online EEP, which are electronic 
educational-methodical sets of disciplines. 15 online EEPs with physics and astronomy as subjects 
were evaluated (Figures 2.3 show the grade point averages by groups of indicators) by teachers 
and students of school as experts. 


This technique revealed the most problematic aspects of online EEPs’ creation and 
exploitation: the lack of organizational and methodological support for studying EEP and ways of 
encouraging students to study (the average score for this component is 1.38), as well as low quality 
of EEP support (this component’s average score is 1.63). 





2,00 1,87 
1,70 1,78 
1,50 + 1,38 
1,00 
0,50 + - 
0,00 + . r . . r : 
Steoctural-Target The group of indicators that Group of EEP’s Informational. Group of indicators that characterize ‘ 
Indicators Group describe the content quality Methodological and EEP contro! functions 


Organizational Support Indicators 


Figure 2.1 — Average marks by groups of indicators 
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Figure 2.3 — Average marks by groups of indicators 
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Figure 3 — Average marks by groups of indicators 


Analyzing the results of experimental work by components separately brings us to the 
following conclusions: 


a) EEP’s informational and methodological materials for studying course 
quantity is insufficient or zero. EEP doesn’t encourage students to study; 

b) EEP is being developed separately from the course’s work program. 
Therefore, structure and content of EEP and work program requirements are 
mismatched; 

c) Teachers do not structure EEP thoroughly and in modules. They lay out 
theoretical and practical parts separately; 

d) insufficiency of interactive forms for learning and control; 

e) Most teachers consider EEP as a way of providing educational material. 


However, they don’t consider EEP as interactive communication with students as 
evidenced by: 


1) Low EEP attendance (“reduction” of material which brings to irrelevance 
of the course); 

2) Insufficient relevance of interactive forms of training and control. Task 
checking and journaling of students’ academic achievements are conducted 
outside the LMS system; 

3) Low level of information support for students in education. 


As a result, one of the main areas suitable for improving the quality of student learning 
is electronic learning which is focused on active independent work of students and includes using 
ICT and multimedia in both traditional and distance learning. 


6. Conclusion 


The quality of e-learning depends directly on the EEPs’ quality, the relevance and 
correctness of the information provided in EEP, as well as the possibilities of organizing control 
and self-control, communication, and complying with health-caring education technologies. 


The presented system of indicators and criteria, as well as a technique for evaluating 
the quality of online EEP allows: 
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(a) Evaluating the components and analyzing the quality of courses. Depending on the 
results of evaluation the expert group or author may take a balanced and reasoned decision on 
accepting, rejecting or revising the proposed publication; 


(b) Summarizing the results and highlighting the main problem areas of EEP 
development throughout its life cycle. The results will enhance the quality of EEPs. This set of 
requirements for EEP’s functional properties implementation will give users the fullest 
opportunity possible to use the software properties of the shell in which the EEP will be 
reproduced; 


(c) Using the evaluation results to make appropriate management decisions on 
improving the EEP’s quality. 


In order to make a decision on the general rating of the publication it is desirable to 
use the following criteria: 


1) Score between 2.5 and 3 is high. No corrections are required and EEP can 
be used in the educational process; 


2) Score between 2 and 2.5 is higher than the average. Before using EEP in 
the educational process, it is necessary to finalize the individual details; 


3) Score between 1.5 and 2 is average. It is necessary to finalize the functional 
properties of EEP and send it back for reviewing; 


4) Score between 1 and 1.5 is below average. EEP requires serious work on 
errors and flaws; 


5) Score between 0 and 1 is low. EEP is not suitable for training. 
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